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ABSTRACT 



A new approach for software debugging, verification and 
validation is disclosed. The present invention utilizes a 
knowledge-based reasoning approach to build a functional 
model of the software code for identifying and isolating 
failures in the software code. The knowledge-based reason- 
ing approach of the present invention uses the software 
design, which is preferably based upon a flow chart or block 
diagram representation of the software functionality, to build 
the functional model. The software block diagram contrib- 
utes to the functional model by defining the inputs and 
outputs of the various blocks of code, as well as defining 
data interconnections between the various blocks of code. In 
accordance with a method of the present invention, test 
points are strategically inserted throughout the code, and 
each test point is associated with a corresponding block of 
code. Expected values of the test points for an expected 
proper-operation execution of the computer program are 
generated. The computer program is then executed on a 
computer, and the actual values of the test points from the 
program execution are compared with the expected values of 
the test points. Failed test points which do not agree with 
corresponding expected values are determined. r ITie func- 
tional model, which includes information functionally relat- 
ing the various test points to one another, is then used to 
isolate the failed test points to one or more sources of failure 
in the code. 

43 Claims, 12 Drawing Sheets 
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METHOD AND APPARATUS FOR lion to other available techniques, is typically implemented 

DEBUGGING, VERIFYING AND to ensure that the software will operate properly under all 

VALIDATING COMPUTER SOFTWARE operational modes. Proper functionality of the software code 

should include performance reliability and robust operation. 

BACKGROUND OF THE INVENTION 5 Efforts continue in the prior art for enhancing the efficiency 

r* * anc * e ® cac y °f software debugging, verification and valid a - 

1, Held ot the Invention Uon procedureSt which typically must be implemented both 
The present invention relates generally to computers and, during and after the design phase of the software is corn- 
more particularly, to computer software debugging, verify- pleted. 

ing and validating devices and techniques. jq 

2. Description of Related Art SUMMARY OF THE INVENTION 

The process of correcting developmental errors within met hod and apparatus of the present invention 
computer software code is commonly referred to as debug- present new approaches for software debugging, verification 
ging. A typical debugging procedure entails a software and validation. The present invention utilizes a knowledge- 
designer executing the prototype software and attempting to 15 based reasoning approach t0 bu n d a functional model of the 
determine which portions of the code are responsible for software code for identifying and isolating failures in the 
incorrect operation. The software developer typically exam- software code. The knowledge-based reasoning approach of 
ines a failure within the software code and attempts to the present invention uses the software design, which is 
determine the portion, module or line of software code that preferably based upon a flow chart or block diagram repre- 
is responsible for the failure. 20 xmxion of the software functionality, to build the func- 

Conventional debugging techniques include steps of tional model. The software block diagram contributes to the 

inserting output statements after subsections of code to functional model by defining the inputs and outputs of the 

notify the developer of the results either before, during or various blocks of code, as well as defining data intercon- 

after the execution of the remainder of the program. By nections between the various blocks of code. In accordance 

incorporating output statements into the code, the developer 25 with a method of the present invention, test points are 

can more quickly deduce the part or parts of code that are strategically inserted throughout the code, wherein each test 

yielding the wrong in -process answer, thereby achieving . point is associated with a corresponding block of code. A 

fault isolation to the respective subsection of code. The standard case is used to determine the expected values of the 

software developer frequently employees a standard case to test points for an expected proper-operation execution of the 

test and debug the code. Using a standard case, the software 30 software code. 

developer can test each result from each output statement in software code is then executed, and the actual values 

the code for accuracy, by comparing each result with an of the test points from the prog ram execution are compared 

expected result for the standard case. wiih the expected values of the test points. Failed test points 

Although the process of debugging has been automated to 35 which do not agree with corresponding expected values are 

some extent through the use of error codes and commercial thus determined. The functional model, which includes 

debuggers for popular development languages, the process information functionally relating the various test points to 

of debugging remains a very time consuming and costly part one another, is then used to isolate the failed test points to 

of the software development process. one or more conclusive sources of failure within the code. 

Even after the software is developed, it must be verified 40 In accordance with one aspect of the present invention, a 

and validated. Software verification and validation generally method of selecting a source failure test point from a 

encompasses an acceptance process in which the software is plurality of test points in a computer program comprises an 

functionally verified to be correct when its operation is initial step of providing a plurality of test points in the 

compared to the required performance defined in the design computer program, and a subsequent step of defining at least 

specification. Operational characteristics of the software arc 45 one fault propagation path. (In another aspect of the present 

verified within the bounds of the development specification. invention, the fault propagation path is referred to as a data 

The verification and validation step is typically implemented propagation path.) The test points are placed into the code of 

both internally to the software development establishment the computer program, and an order of data flow among the 

and by the ultimate customer, such as the federal govern- test points is determined. The order of data flow defines at 

ment. For example, the U.S. Department of Defense may 50 least one fault propagation path through the plurality of test 

have a software product delivered to control a missile. The points. The source failure test point is defined as having a 

U.S. Department of Defense in this instance would obvi- highest probability relative to other test points in the com- 

ously be interested in conducting an independent verification puter program of being a source of failure. The at least one 

of the functionality of the software. fault propagation path associates at least two of the plurality 

The process of verifying the operation of software under 55 of test points and an order of data flow and data dependency 

all operational modes is typically a very time-consuming within the computer program. 

and costly effort, in which all possible scenarios are The method of the present invention includes a step of 
executed for the software, to identify its response under all generating a standard case for the test points of the computer 
conditions. The conventional approach comprises a user program in which expected values of the test points for an 
stepping through operational modes and measuring a soft- 60 expected proper-operation execution of the computer pro- 
ware's responses. By doing so, the user can verify the gram are generated. The computer program is then executed 
software's functionality with some certainty, thereby vali- on a computer, and the actual values of the test points from 
dating it for operation in the intended application. The the program execution are compared with the expected 
amount of effort required for software verification and values of the test points. Failed test points which do not 
validation is typically augmented by a user implementing 65 agree with corresponding expected values are determined, 
extensive examination of each line of the software code. An additional step is provided for finding the source failure 
This line-by-line examination of the software code, in addi- test point that is earliest (relative to other failure test points 
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in the at least one fault propagation path) in an order of data FIGS. 6-9 comprise flow charts illustrating a method in 

flow and data dependency. The failure test points are asso- accordance with an alternative embodiment of the present 

ciated with the at least one data propagation path, and the invention; and 

step of finding the source failure test point that is earliest FIG. 10 is a flow chart illustrating a method in accordance 

inc udes a step of selecting the failed test point which is 5 ^ anQlher alternative embodime nt of the present inven- 

earliest in the at least one fault propagation path. ^ Qn 

Another aspect of the present invention includes a method 

of determining a source of failure test point from a plurality DETAILED DESCRIPTION OF THE 

of test points in a computer program. The source of failure PRESENTLY PREFERRED EMBODIMENTS 

test point has a highest probability relative to other test 30 

points in the computer program of being a source of failure. Referring more particularly to the drawings, FIG. 1 illus- 

The method includes a step of determining a sequential flow trates a computer system 20 comprising a microprocessor 

of data among a plurality of test points in the computer 22 > a Random Access Memory (RAM) 24 and a Read Only 

program, and a step of ranking the plurality of test points, Memory (ROM) 26. The computer system 20 of the pres- 

using the determined sequential flow of data, in an order of ently preferred embodiment further comprises an input 

an earliest test point in the determined sequential flow of 15 device 28, such as a keyboard, and Input/Output (I/O) device 

data to a last test point in the determined sequential flow of 30 which may be connected to a printer, for example, and a 

data, to thereby generate a ranked set of test points. (In display 33. A system bus 35 interconnects the various 

another aspect of the present invention, the ranked set of components. Other components and buses (not shown) may 

data is generated by ranking the plurality of test points in be connected to the system bus 35 for providing additional 

accordance with an order of execution and data dependency functionality and processing capabilities. A network access 

of the test points.) device may be included, for example, for providing the 

The method of the present invention includes a step of computer system 20 access to an Internet, intranet or other 
generating expected values of the test points for an expected network system. The computer system 20 of the present 
proper-operation execution of the computer program. The 25 invention is configured to implement and/or facilitate the 
computer program is then executed on a computer, and the implementation of all of the below-described processes, 
actual values of the test points from the program execution which m described in connection with FIGS. 2-10. 
are compared with the expected values of the test points. In accordance with the present invention, the computer 
Failed test points which do not agree with corresponding system 20 is used as a powerful diagnostic tool for facili- 
expected values are determined. Each failed test point indi- 30 tating software debugging, verification and validation pro- 
cates an erroneous program operation or result. The method cedures. A knowledge-based reasoning approach is used in 
continues with a step of determining a failed test point from accordance with the present invention to build a functional 
the plurality of failed test points that ranks earliest among model of the software code for identifying and isolating 
failed test points in the ranked set of test points. The failures in the software code. The knowledge-based reason- 
earliest-ranked failed test point is determined to be the 35 ing approach of the present invention uses the software 
source failure test point. design, which is preferably based upon a flow chart or block 

The step of arranging the plurality of test points can diagram representation of the software functionality, to build 

include a step of defining at least one fault propagation path. tne functional model. After the functional model is built, the 

The plurality of failed test points can correspond to a single functional model is input into the computer system 20. The 

ranked group of test points which defines a fault dependency 40 software block diagram representing the software code 

set. In another aspect of the present invention, the plurality contributes to the functional model by defining the inputs 

of failed test points corresponds to a plurality of ranked amJ outputs of the various blocks of code, as well as by 

groups of test points, with each ranked group of lest points defining data interconnections between the various blocks of 

defining a separate fault dependency set. code. 

The present invention, together with additional features 45 ln accordance with a method of the present invention, test 

and advantages thereof, may best be understood by reference points are strategically inserted throughout the code, and 

to the following description taken in connection with the each test point is associated with a corresponding block of 

accompanying illustrative drawings. code. A standard case, or "test*' case, is run for the code, and 

BRIEF DESCRIPTION OF THE DRAWINGS 



failed test points are determined for the standard case. The 



50 functional model, which includes information functionally 

FIG. 1 is a system block diagram illustrating a number of relating the various test points to one another, is then used 

components in accordance with an embodiment of the to isolate the failed test points to one or more sources of 

present invention; failure in the code. 

FIG. 2 is a flow chart illustrating the interconnectivity and i n the following description, a method of the present 

process flow among a number of blocks of code of a first 55 invention is illustrated in the context of a simple example, 

exemplary computer program in accordance with the present with reference to FIG. 2. After the example, the method of 

invention; the presently preferred embodiment is disclosed in greater 

FIGS. 3a-3d comprise a flow chart illustrating the method detail with reference to the flow charts of FIGS. 3a-3d. The 

of the presently preferred embodiment; following FIGS. 4 and 5, and the accompanying text asso- 

FIG. 4 is a flow chart illustrating the interconnectivity and 60 ciated therewith, disclose examples of the method of FIGS, 

process flow among a number of blocks of code of a second 3a-3rf applied to particular arrangements of program code, 

example computer program in accordance with the present FIGS. 6-10 discuss additional embodiments of the present 

invention; invention. 

FIG. 5 is a flow chart illustrating the interconnectivity and The below set of Fortran code comprises a main Program 

process flow among a number of blocks of code of a third 65 "ABC that calls a Subroutine "NONO." The subroutine 

example computer program in accordance with the present CALL statement in the main program comprises two argu- 

invention; ments "1.0" and "X," and the actual subroutine has an 
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argument list of "A" and "B." A problem exists in the 
subroutine. Namely, the constant "1.0" in the CALL state- 
ment of the main program is overwritten with a value of "2" 
in the subroutine. As a result of this error, occurrences of 
"1.0** in the code after execution of the subroutine are 
considered by the processor as having a value of "2.0." The 
set of Fortran code is set forth below in Text Layout 1. 



program. A third test point in the form of a "WRITE 
TESTDATA; 3, 1.0" command of placed after the RETURN 
statement in the subroutine to test whether the value of "1.0" 
is being interpreted correctly during program execution. The 
test points are illustrated inserted into the program code 
below in Text Layout 3. 



PROGRAM ABC 

CALL NONO(1.0,X) 

PRINT", *1 X, 1.0 + X 

STOP 

END 

SUBROUTINE NONO(A,B) ' 

B - 1.0 

A - 2.0 

RETURN 

END 

Text Layout 1. 



PROGRAM ABC 
10 CALLNONO(IAX) 

PRINT*, '1 +', X, 1.0 + X 

STOP 

END 

SUBROUTINE NONO(A,B) 
B - 1.0 
15 A -2.0 

WRITE TESTDATA; 1, Yes 
WRITE TESTDATA; 2, A 
RETURN 

WRITE TESTDATA; 3, 1.0 
END 



In accordance with the method of the present invention, a 
user first groups the lines of code of the above-program into 
functional blocks of code. The set of Fortran code is shown 
below partitioned into three functional blocks of code. In 
particular, the subroutine CALL statement in the main 
program is designated as the first block; the print statement 
in the main program is designated as the second block; and 
the set of code comprising the subroutine is designated as the 
third block, as indicated below in Text Layout 2. 



PROGRAM ABC 
CALL NONO (1.0,X) 
PRINT*, '1 +\ X, 1.0 + X 
STOP 
END 

SUBROUTINE NONO(A,B) 
B- 1.0 
A -2.0 
RETURN 
END 

Text Layout 2. 



pesignate as first block 
pesignate as second block 



[Designate as third block 



I 



50 



The inputs and outputs of each of the functional blocks of 
code are then identified by the user. In the presently pre- 
ferred embodiment, a flow chart or functional block diagram 
representation of the program is generated to achieve this 
end. The flow chart, which may already exist, preferably 
defines the processing flow through the various blocks of 
code. The flow chart further preferably defines inputs and 
outputs of the various blocks of code, and also defines the 
data interconnections between the various blocks of code. 

Turning to FIG. 2, the first block 40 outputs data to the 
third block 42 and, in turn, the third block 42 inputs data 
from the first block 40. Similarly, the third block 42 outputs 
data to the second block 44, and the second block 44 inputs 55 
data from the third block 42. 

Test points are then written into the Fortran code. The test 
points arc preferably placed at outputs of the various blocks 
of code, but may be placed elsewhere in alternative embodi- 
ments. In the present example, three test points are placed at 60 
the output of the third block 42. In particular, a first test point 
in the form of a statement "WRITE TESTDATA; 1, Yes" is 
placed before the RETURN statement in the subroutine to 
test whether the subroutine is ever executed, and a second 
test point in the form of a "WRITE TESTDATA; 2, A" 
command is placed after the first test point to test whether 
the subroutine passes the expected value back to the main 



65 



pesignate as first block 
]Dcsignatc as second block 



Designate as third block 



20 



Text Layout 3. 



After the interconnectivity among the various blocks of 
code has been established with a flow chart and test points 
have been incorporated into the code, the user identifies 

25 dependency sets among the various test points within the 
block diagram. Each dependency set represents a data propa- 
gation path through a number of test points. Since the flow 
chart of FIG. 2 only has three serial blocks, only one 
dependency set exists, which includes all three test points. 

30 As shown below in Text Layout 4, a single dependency set 
matrix is generated for the present example, comprising test 
points 1, 2 and 3. 



35 



DEPENDENCY SET MATRPC 
Test Point Set Set Includes Test Points 



1, 2 and 3 



40 



Text Layout 4. 



A test point mapping matrix is generated next, which 
maps each test point with the corresponding block of code 
that it is testing. In the present example, all three of the test 
points are designed to test the third block of code. The test 
point mapping matrix is provided below in Text Layout 5. 



TEST POINI' MAPPING MATRIX 


Test Point 


Block of Code 


1 


3 


2 


3 


3 


3 




Text layout 5. 



In the presently preferred embodiment, after the user has 
established the flow chart, the placement of test points, the 
dependency set matrix, and the test point mapping matrix, 
the user selects a known "test" case and determines the 
values for the various test points. A hypothetical execution 
of the Fortran program of the present example for a standard 
case where desired results are achieved, would yield an 
output of "yes" for the first test point, an output of "2" for 
the second test point, and an output of "1.0" for the third test 
point. These three outputs represent what a user would 
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expect from an execution of the Fortran code. The three 
outputs are shown in Text Layout 6 tabulated in a reference 
test point data file. 



RUN TIME PASS/FAIL MATRIX 



REFERENCE TEST POINT DATA FILE 



Reference Answer 



Test Point 



Pass/Fail 



Yes 
2 

1.0 



Text Layout 6. 



10 



IS 



The user next executes the program, using inputs for the 
known test case. Upon execution of the program, the pro- 
gram writes output data to the run time test point data file at 
the three test points. The below Text Layout 7 illustrates that 
a value of "yes" is output by the program for the first test 
point, a value of "2" is output by the program for the second 
test point, and a value of "1.0" is output by the program for 
the third test point. 



RUN TIME TEST POINT DATA FILE 


Test Point 


Reference Answer 


1 


Yes 


2 


2 


3 


1.0 




Text Layout 7. 



The method of the present invention, upon obtaining 
values for the run time test point data file, correlates the 
reference test point data file with the runtime test point data 
file, and creates the run time pass/fail matrix. If the run time 
result (output data) for the test point is as predicted by the 
reference test point data file, then the method of the present 
invention enters a pass status in the run time pass/fail matrix. 
If the run time result for the test point is different than that 
predicted by the reference test point data file, then the 
method of the present invention enters a fail status in the run 
time pass/fail matrix. 

As indicated in Text Layout 8 below, since the output for 
the first test point is the same for both the expected result (as 
indicated in the reference test point data file) and the actual 
result (as indicated in the run time test point data file), the 
first test point is assigned a pass status. Since the output for 
the second test point is the same for both the expected result 
(as indicated in the reference test point data file) and the 
actual result (as indicated in the run time test point data file), 
the second test point is assigned a status of pass. The output 
value for the third test point in the run lime test point data 
file is different than the output value for the third test point 
in the reference test point data file. Accordingly, the third test 
point is assigned a status of fail. In alternative embodiments, 
a fail status is only assigned to test points which have actual 
output values beyond a predetermined tolerance, relative to 
an output value in the reference test point data file for the test 
point. 



20 



Text Layout 8. 



The run time pass/fail matrix indicates which test points 
have failed. The mere occurrence of a test point failure, 
however, does not conclusively point to the source of the 
error in the program code. 

In accordance with an important feature of the present 
invention, each failed test point is correlated with one or 
more dependency sets, to which the failed test point belongs. 
The method of the present invention references the depen- 
25 dency set matrix to determine the particular dependency sets 
that contain failed test points. The particular dependency 
sets that contain failed test points are assigned a fail status 
in a fault detection set matrix, as shown in Text Layout 9. 

30 

FAULT DETECTION SET MATRIX 



35 



40 



Fault 
Detection Set 



Set Includes 
Test Points 



Set 
Pass/Fail 



3, 2 and 1 
Text Layout 9. 



The method of the present invention searches the fault 
detection set matrix for failed dependency sets that contain 
the same test points, and creates the fault isolation set 
4 5 matrix. The fault isolation set matrix comprises a first 
column indicating the fault isolation set number, a second 
column indicating the test points included in the fault 
isolation set, and a third column indicating the block or 
blocks of code that contain the error. In the present example, 
as illustrated in Text Layout 10, a single fault isolation set 
includes the third test point. 

Since the third test point is designed to test the third block 
of code, the third column of the fault isolation matrix should 
be able to indicate that an error in the code of the main 
program is present. Because the present example comprises 
a serial flow chart diagram, however, the method of the 
present invention cannot isolate to one of the three blocks of 
code if only the third block of code in the series fails. Here, 
the first test point passed indicating that the subroutine 
executed, and the second test point passed indicating that the 
subroutine passed the expected value back to the main 
program. Therefore, the present example indicates that a 
65 problem exists in the main program. Upon examination of 
the code by a user, a correction to the third block of code can 
be implemented. 



50 



55 



60 
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FAULT ISOLATION MATRIX 

Fault Set Includes Code 

Isolation Set Test Points Block 



1 3 3 

Text Layout 10. 



The method of the presently preferred embodiment is 
depicted in the process diagrams of FIGS. 3a-3d. In accor- 
dance with the present invention, a method for automating 
and increasing the accuracy of software debugging, verifi- 
cation and validation is disclosed. In one presently preferred 
embodiment, the method employees logic and set-covering 
theory to matrix data. In contrast to typical prior- art devices, 
the method of the presently preferred embodiment does not 
rely on a user and a user's particular expertise during code 
execution and debug. The method of the presently preferred 
embodiment only needs to know the structure of the design 
and the test point results from a run of the program structure 
against a known problem. 

The method of the presently preferred embodiment uses 
strategically placed test points within the code of the pro- 
gram that yield output data when the program is executed. 
This output data is used to deduce the location or locations 
of the most probable programming errors. Each test point 
comprises an intermediate output statement placed within 
the code to determine program execution status at a particu- 
lar stage of execution of the program. The test points 
typically comprise data outputs or inputs to given software 
processes or functions. The method of the presently pre- 
ferred embodiment uses the interdependency of various 
blocks of code to isolate the most probable failure of the 
code execution to one block or a group of blocks of code, 
depending on the number and placement of test points 
within the code. 

The process of the presently preferred embodiment begins 
at step S100, and advances to step S102 where lines of code 
of the program to be tested are grouped into functional 
blocks. At step S104, the user identifies inputs and outputs 
for each individual block of code. The knowledge-based 
reasoning approach of the present invention uses the soft- 
ware design, which is preferably in the form of a flow chart, 
to build a functional model of the software code for iden- 
tifying and isolating failures in the software code. From the 
flow chart, the method of the present invention derives the 
interconnectivity of inputs and outputs among the blocks of 
code. 

At step S106, a run time test point data file is defined for 
later use. Output data from the lest points will be written to 
the run time test point data file during execution of the 
program. In alternative embodiments, the run time test point 
data file may be defined immediately before use or at any 
other time before actual use thereof. 

Test points are inserted into the code at step S108, Each 
test point instructs the program during execution to write 
output data to the run time test point data file. From a basic 
flow chart, a diagram can be generated which expresses the 
overall program into code blocks and interconnections of 
flow connectivity among the code blocks. The dependency 
of outputs on inputs for each code block provides a design 
knowledge-base, which allows for later analysis against a 
test case. In the presently preferred embodiment, for optimal 
overall results, at least one input and output are identified for 
each code block in the flow chart. 
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As presently preferred, a test point output statement is 
implanted into the code at each output of a code block, to 
provide for high testability of the design. The method of the 
present invention is capable of detecting multiple, simulla- 

5 neously occurring code faults, provided that in adequate 
number of test points for the method to distinguish between 
fault propagation paths is provided within the code. If a user 
uses fewer test points, the method of the present invention 
will still function properly, but may not be able to isolate the 

10 fault to a single block of code. Even if a fewer number of test 
points is used, proper selection of the blocks of code (i.e. 
good fidelity of visibility into the design and fewer lines of 
code per block) may offset the absence of test points at every 
block of code output. One objective of the present invention 

1S is to isolate the fault to the smallest group of code so that the 
user may find and fix the problem faster. 

The method of the present invention provides a structural- 
based modeling approach, as opposed to a functional 
approach, since the method of the present invention does not 

20 need to know the actual content or function of each block of 
code. 

A block diagram or flow chart of the program is generated 
at step S110, if the flow chart has not already been previ- 
ously established. The flow chart should show the intercon- 

25 nectivity among the various blocks of code, including infor- 
mation pertaining to the outputs which each block of code 
receives from other blocks of code as inputs. At step S113, 
dependency sets in the flow chart are identified. The depen- 
dency sets are placed into a dependency set matrix at step 

30 S116. Each dependency set defines a flow of data and/or 
program operation through a number of blocks of code. By 
definition, any block of code located "downstream" (later in 
execution), relative to another block of code, is dependent 
on the other block of code, if the downstream block of code 

35 receives and relies upon data from the upstream block of 
code. In accordance with the presently preferred embodi- 
ment, execution dependency of program code blocks is 
modeled using set theory. A dependency set contains as its 
contents test points that are interdependent. For example, a 

40 dependency set can be defined as all test points which are 
dependent on a particular block output. 

At step S118, a standard or test case, where the correct 
results for each test point arc available, is used to determine 
the expected values of the test points for an expected 

45 proper-operation execution of the software code. In the 
presently preferred embodiment, a user applies a known case 
and predicts what the test point outputs should be for a "no 
fault" case. The predicted testpoint output data, which is 
determined for each test point (from a known, correct 

50 solution) for the test case, is placed into a reference test point 
data file at step S120. 

A test point mapping matrix is defined at Step S124 for 
later use. The test point mapping matrix may be used in the 

5S final analysis (step SI 48) to associate each test point with the 
block of code that it is testing. In alternative embodiments, 
the test point mapping matrix may be defined immediately 
before use or at any other time before actual use thereof. 
A controlled execution for the known case of the com- 

60 puter code being analyzed is initiated at step S127. Upon 
execution of the program for the known case, the method of 
the present invention first writes data to the run time test 
point data file at step S129. 
The method correlates the reference test point data file 

65 with the run lime test point data file at step S131, to create 
the run time pass/fail matrix at step S135. Taking into 
account the operational mode of the software at the instant 
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of data acquisition, a pass or a fail status is assigned for each comprises a first column indicating the fault isolation set 

test point during the program execution. This is achieved, for number, a second column indicating the test points included 

example, by evaluating the performance of state data from in the fault isolation set, and a third column indicating the 

each test point, and by determining if that performance is block or blocks of code that contain the error. Each row in 

within a predetermined acceptable tolerance for the specific 5 the fault isolation set matrix represents a source of failure, 

operational mode. If the runtime result, written during When program operation branches directly to step S150, the 

execution of the program by a given test point, is as number of the test point that failed will be the same as the 

predicted by the reference test point data file, the method of number of the block of code containing the error. That is, 

the present invention enters a pass status into the run time columns two and three will have the same content in the 

pass/fail matrix. If the runtime result is not as predicted by 3Q fault isolation set matrix. 

the reference test point data file, the method of the present , f> 0D the on the other haod> the number of lest ints does 

invention enters a fa, 1 status into the run time pass/fail not directl amspoad to the number of blocks of code, and 

matrix At step S137, the method determines whether each ^ ^ number of eacfa ^ mt ^ UQ{ directl c ond 

given test point has a pass or a fail status and enters the numbef £ cq6q m ( tests / then 

appropriate value into the run time pass/fail matnx. . . . t „~ AO r ' . 

V. .... - r . . . . 15 program operation branches over to step S148. For example, 

By monitoring the data from each test point in real-time if the user has assigned more test points than blocks of code, 

during execution of the program code^he performance of as ^ ^ ^ cage wfaen Qne 0f more b]ocks Qf ^ have 

the program code can be evaluated. The data originating ^ ^ method references ^ ^ 

from the test points in real-time is compared with the results . r . ■ , *\ . , c , {\ . 

from the known test case. In steps S139 to S160, discussed malnx l ° d f" nine wt ! ich ™ 0ck f 0 ^ f ach test P 0Ult 

below, the process of the present invention is able to fault 20 * associated. Subsequently the fault isolation matrix, 

isolate the incorrect result or results to the specific block or mdud »* column lhree lhereof ' 15 Populated at step S150. 

blocks of the flow chart, to thereby fault isolate to the At step S153, the method of the present invention inspects 

specific group of lines or line of code which is incorrect. lhe fault isolation set matrix to determine whether any 

Since all data and the functional model are preferably entries exist therein. If the fault isolation set matrix is empty, 

located on a computer, the process of the present invention 25 tnen tne method ends. 

is completely automated in one embodiment, to increase the If, on the other hand, one or more entries exist in the fault 

accuracy and decrease the time required to isolate a software isolation set matrix, the program operation branches to step 

failure. S155. The fault isolation set matrix and the code are 

Having acquired the run time pass/fail matrix from step examined for errors in step S155, and any changes are 

S137, the method of the present invention now has infor- 30 incorporated into the failed blocks of code at step S160. The 

mation regarding the test points in the program code that process branches back to step S102 and the process repeats, 

have failed. At step S139 the method of the present invention until lne fault isolation set matrix is empty at the inspection 

sets out to determine all of the dependency sets that contain ste P S153, indicating a successful program execution with 

failed test points. Each test point with a fail status is no code failures. 

correlated to the dependency set or sets which contain the 35 In a simple embodiment of the present invention, all steps 

failed-status test point, to enable a determination as to which except for step S127 and S129 can be performed manually, 

dependency sets contain failures. The method collects all of In another embodiment, all steps except for steps S127 to 

the dependency sets (from the dependency set matrix) that S153 are performed manually. In still other embodiments, 

contain failed-status test points, as indicated by the run time steps S102 to S160 are all automatically performed on a 

pass/fail matrix. At step S141 the method of the present 40 computer. 

invention defines the fault detection set matrix and places all Turning now to FIG. 4, the method of the presently 

dependency sets that contain failed-status test points therein. preferred embodiment is discussed with continued reference 

As presently embodied, the method uses set covering to FIGS. 3a-3d. The method of the present invention first 

theory and associated logic to correlate fault dependency 45 groups the lines of code of a program into functional blocks 

among several matrices containing the necessary data to of code (step S102), and the inputs and outputs of each of the 

isolate the fault or faults to one or more blocks of code. In functional blocks of code are identified (step S104). The 

accordance with a preferred embodiment of the present flow chart or functional block diagram representation of the 

invention, test point commonality among the dependency program is generated to achieve this end, as illustrated in 

sets, which are known to contain test point failures, provides 5Q FIG. 4. The flow chart defines the processing flow through 

one effective means of fault isolation. In the presently the various blocks of code, defines inputs and outputs of the 

preferred embodiment, the contents of the fault detection set various blocks of code, and also defines the data intercon- 

matrix are scanned to determine test point commonality and nections between the various blocks of code, 

to ultimately produce the fault isolation set matrix, which is The processing flow in the functional block diagram of 

the output of the method of the present invention. 55 F!G . 4 branches from a program input 180 to a first block 

In step 143, the method searches the fault detection set 183. The processing flow in the functional block diagram 

matrix for sets that contain the same failed test points. The passes from the first block 183 to the second block 186 and, 

method of the present invention determines whether the subsequently, passes from the second block 186 to the fifth 

number of test points directly corresponds to the number of block 197. In addition to branching to the first block 183, the 

blocks of code at step S145. If the number of test points 60 processing flow from the program input 180 also branches to 

directly corresponds to the number of blocks of code, and if the fourth block 193. Processing flow in the functional block 

the number of each test point directly corresponds to the diagram passes from the fourth block 193 to both the second 

number of the block of code it tests, then program operation block 186 and the third block 189 and, subsequently, passes 

branches down to step S150. from the third block 189 to the fifth block 197. 

In step S150, the fault isolation set matrix is defined. As 65 The run time test point data file can be defined at this point 

presently embodied, the fault isolation set matrix contains or later (step S106), and test points are incorporated into the 

three columns. In particular, the fault isolation set matrix program code (step S108) to write output data into the run 
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time test point data file. The test points are preferably placed 
at outputs of the various blocks of code, but may be placed 
elsewhere in alternative, but not equivalent, embodiments. 

After the interconnectivity among the various blocks of 
code has been established with a flow chart (step S110) and 
test points have been incorporated into the code, dependency 
sets among the various test points within the block diagram 
are identified (step S113), and the identified dependency sets 
are placed into a dependency set matrix (step S116), Each 
dependency set represents a data propagation path through a 
number of test points. As shown below in Text Layout 11, 
six dependency set matrices are generated for the present 
example. 



DEPENDENCY SET MATRIX 
Test Point Set Set Includes Test Points 



5, 2 and 1 
5, 2 and 4 
5, 3 and 4 

2 and 1 

3 and 4 
2 and 4 



10 



35 



20 



14 



matrix. If the run time result for the test point is different 
than that predicted by the reference test point data file, then 
the method of the present invention enters a fail status into 
the run time pass/fail matrix (step S137). 

As indicated in Text Layout 13 below, since the output for 
the first test point is the same for both the expected result (as 
indicated in the reference test point data file) and the actual 
result (as indicated in the run time test point data file), the 
first test point is assigned a pass status. Since the output for 
the second test point is different than both the expected result 
(as indicated in the reference test point data file) and the 
actual result (as indicated in the run time test point data file), 
the second test point is assigned a fail status. As another 
example, since the output value for the third test point in the 
run time test point data file is the same as the output value 
for the third test point in the reference test point data file, the 
third test point is assigned a status of pass. In alternative 
embodiments, a fail status is only assigned to test points 
which have actual output values beyond a predetermined 
tolerance, relative to an output value in the reference test 
point data file for the test point. The run time pass/fail matrix 
for the present example is provided below for all five test 
points. 



Text Layout 11. 



25 



Each dependency set includes a different group of all test 
points that are dependent upon one another. For example, 
one dependency set includes all test points associated with 
the fifth block 197, the second block 186 and the first block 30 
183. Another dependency set includes all test points asso- 
ciated with the fifth block 197, the third block 189 and the 
fourth block 193. 

After the flow chart, the placement of test points, and the 
dependency set matrix are all established, a known "test" 35 
case is selected where the values for the various test points 
are available or can be determined. The reference test point 
data file is established (steps S118 and S120), and the values 
for the various test points are entered therein. 

40 

Each test point is mapped with the corresponding block of 
code that it is testing (step S122), and a test point mapping 
matrix is generated (step S124), The test point mapping 
matrix is provided below in Text Layout 12. 



TEST POINT MAPPING MATRIX 
Test Point Block of Code 



2 2 

3 3 

4 4 

5 5 
Text Layout 12. 

. 55 

The program is then executed, using inputs for the known 
test case (step S127). During execution of the program, the 
program writes output data from the test points to the run 
time test point data file (step S129). The method of the 60 
present invention, upon obtaining values for the run time test 
point data file, correlates the reference test point data file 
with the runtime test point data file (step S131), and creates 
the run time pass/fail matrix (step S135). If the run time 
result (output data) for the test point is as predicted by the 65 
reference test point data file, then the method of the present 
invention enters a pass status into the run time pass/fail 



RUN TIME PASS/FAIL MATRIX 

Test Point Pass/Fail 

1 P 

2 F 

3 P 

4 P 

5 F 
Text Layout 13. 



The run time pass/fail matrix indicates which test points 
have failed. The mere occurrence of a test point failure, 
however, does not conclusively point to the source of the 
error in the program code. In accordance with the present 
invention, the mere process of fault detection is carried an 
additional step to fault isolation. Fault detection is simply 
the knowledge that a fault has occurred, without the addi- 
tional knowledge of where or how the fault has occurred. 
Fault isolation, in accordance with the present invention, is 
the additional process by which the source of location of the 
fault is determined. The method of the present invention first 
detects a fault or faults, and then isolates the fault or faults 
to a section of code. The lowest possible section of code, in 
accordance with the presently preferred embodiment, is a 
block of code. 

In accordance with an important feature of the present 
invention, each failed test point is correlated with one or 
more dependency sets to which the failed test point belongs. 
The method can now deduce, for example, "if test point 5 
failed and it was dependent upon test point 3 and 2, but test 
point 3 passed and test point 2 failed then test point 5 failed 
due to either itself or test point 2 failing," and "if test point 
2's only dependency was on test points 1 and 4 and both of 
them passed, then the fault must have occurred in a block of 
code which has its output connected to test point 2." 

By processing the fault deduction logic "did this depen- 
dency set contain a test point failure?" on all dependency 
sets in the dependency set matrix in successive progression 
from larger sets to smaller sets and then comparing which 
sets have failures, the method converges on a solution that 
shows the smallest fault isolation set. More particularly, the 
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method of the presently preferred embodiment references 
the dependency set matrix to determine the particular depen- 
dency sets that contain failed test points. The particular 
dependency sets that contain failed test points are assigned 
a fail status in a fault detection set matrix (step S141), as 
shown in Text Layout 14. 



FAULT DETECTION SET MATRIX 


Set Pass/Fail 


Fault Detection Set 


Set Includes Test Points 


1 


5, 2 and 1 


F 


2 


5, 2 and 4 


F 


3 


5, 3 and 4 


F 


4 


2 and 1 


F 


5 


3 and 4 


F 


6 


2 and 4 


F 



The method of the present invention searches the fault 
detection set matrix for failed dependency sets that contain 
the same test points (step S143), and creates the fault 
isolation set matrix (step S150). The fault isolation set 
matrix comprises a first column indicating the fault isolation 
set number, a second column indicating the test points 
included in the fault isolation set, and a third column 
indicating the block or blocks of code that contain the error. 
The fault isolation matrix is provided in Text Layout 15. 



FAULT ISOLATION MATRIX 




Fault 


Set Includes 


Code 


Isolation Set 


Test Points 


Block 


1 


2 


2 




Text Layout 15. 
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The result in this example shows that the method was able 
to isolate the fault to a cause in one block of code which had 
its output tested by test point 2. The method, when applied 
to large programs that do not have intuitive, easy to under- 
stand flow diagrams, can provide a powerful analytical tool. 
The method, which can quickly analyze a program in its 
entirety from a single execution of the program and yield 
fault isolation results on individual blocks of code, presents 
new and useful utility, as compared to conventional methods 
which sequentially examine code in a time-consuming fash- 
ion. 

In the present example, test point 2 is found to be a 
conclusive fault. Test point 5, on the other hand, does have 
a failed status but has not been found to necessarily be a 
conclusive error. The method of the presently preferred 
embodiment, as applied to the present example, cannot 
determine with certainty whether lest point 5 is a conclusive 
failure or whether the failure of test point 5 is a propagated 
failure from test point 2. In one embodiment of the present 
invention, a masked fault isolation matrix can be generated 
as an output, comprising test points which have been deter- 
mined to be faults but which have not been determined to be 
conclusive faults. The masked fault isolation matrix for the 
present example is provided below in Text Layout 16. 



45 



MASKED FAULT ISOLATION MATRIX 


Masked 


Set Includes 


Code 


Fault Set 


Test Points 


Block 


1 


5 


5 




Text Layout 16. 
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Turning now to FIG. 5, the method of the presently 
preferred embodiment is discussed as applied to another 
example, with reference again to FIGS. 3a-3d. The method 
groups the lines of code of a program into functional blocks 
15 of code (step S102), and the inputs and outputs of each of the 
functional blocks of code are identified (step SI 04). The 
flow chart of FIG. 5 is generated to achieve this end. As with 
the example of FIG. 4, the flow chart of FIG. 5 defines the 
processing flow through the various blocks of code, defines 
inputs and outputs of the various blocks of code, and also 
defines the data interconnections between the various blocks 
of code. 

The processing flow in the functional block diagram of 
FIG. 5 branches from a program input 201 to a first block 
202. The processing flow in the functional block diagram 
passes from the first block 202 to the second block 204 and, 
subsequently, passes from the second block 204 to the fifth 
block 210. The processing flow finally moves from the fifth 
block 210 to the ninth block 220. In addition to branching to 
the first block 202, the processing flow from the program 
input 201 also branches to the fourth block 208. Processing 
flow in the functional block diagram passes from the fourth 
block 208 to both the second block 204 and the third block 
206 and, subsequently, passes from the third block 206 to the 
fifth block 210. The processing flow from the program input 
201 further branches to the sixth block 212 and, subse- 
quently, branches to both the seventh block 214 and to the 
eight block 216. Processing flow from the eight block 216 
and from the seventh block 214 converges onto the ninth 
block 220. 

In accordance with the method of the present invention, 
the run time test point data file is next defined (step S106), 
and test points are incorporated into the program code (step 
S108) to write output data into the run time test point data 
file. The test points are preferably placed at outputs of the 
various blocks of code, but may be placed elsewhere, as 
well. 

The interconnectivity among the various blocks of code is 
50 established with a flow chart (step S110) and test points are 
incorporated into the code. Dependency sets among the 
various test points within the block diagram are identified 
(step S113), and the identified dependency sets are placed 
into a dependency set matrix (step S116). Each dependency 
55 set represents a data propagation path through a number of 
test points. As shown below in Text Layout 17, 14 depen- 
dency set matrices arc generated for the present example. 
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DEPENDENCY SET MATRIX 
Test Point Set Set Includes Test Points 



5, 2 and 1 
5, 2 and 4 
5, 3 and 4 
2 and 1 
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-continued 



-continued 



DEPENDENCY SET MATRIX 



Test Point Set 


Set Includes Test Points 


5 


3 and 4 


6 


2 and 4 


7 


5 and 6 


8 


7 and 6 


9 


9, 7 and 6 


10 


9, 5 and 6 


11 


9, 5, 3 and 4 


12 


9, 5, 2 and 1 


13 


9, 5, 2 and 4 


14 


9, 8 and 6 




Text Layout 17. 



A known "test" case is next selected where the values for 
the various test points are available or can be determined. 
The reference test point data file is established (step S120), 
and the values for the various test points are entered therein. 
Each test point is mapped with the corresponding block of 
code that it is testing (step S122), and a a test point mapping 
matrix is generated (step S124). The test point mapping 
matrix is provided below in Text Layout 18. 



10 



RUN TIME PASS/FAIL MATRIX 
Test Point Pass/Fail 



Text Layout 19. 



The run time pass/fail matrix indicates which test points 
is have failed, but does not conclusively point to the source of 
the error in the program code. Each failed test point is 
correlated with one or more dependency sets to which the 
failed test point belongs. The method of the present inven- 
tion references the dependency set matrix to determine the 
20 particular dependency sets that contain failed test points. 
The particular dependency sets that contain failed test points 
are assigned a fail status in a fault detection set matrix (step 
S141), which is shown in Text Layout 20. 
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FAULT DETECTION SET MATRIX 



1 1 

2 2 

3 3 

4 4 

5 5 

6 6 

7 7 

8 8 

9 9 
Text Layout 18, 



Execution of the program is next implemented, in which 
inputs for the known test case are used (step S127). During 
execution of the program, the program writes output data 
from the test points to the run time test point data file (step 
S129). The method of the present invention, upon obtaining 
values for the run time test point data file, correlates the 
reference test point data file with the runtime test point data 
file (step S131), and creates the run time pass/fail matrix 
(step S135). If the run time result (output data) for the test 
point is as predicted by the reference test point data file, then 
the method of the present invention enters a pass status in the 
run lime pass/fail matrix. If the run lime result for the lest 
point is different than that predicted by the reference test 
point data file, then the method of the present invention 
enters a fail status in the run time pass/fail matrix (step 
S137), The run time pass/fail matrix for the present example 
is provided below for all nine test points. 



Fault 


Set Includes 


Set 


Detection Set 


Test Points 


Pass/Fail 
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5, 2 and 1 
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2 


5, 2 and 4 
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3 


5, 3 and 4 


F 
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2 and 1 
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5 


3 and 4 
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2 and 4 


F 
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5 and 6 
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7 and 6 
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9, 7 and 6 
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10 


9, 5 and 6 


F 


11 


9, 5, 3 and 4 
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12 


9, 5, 2 and 1 


F 


13 


9, 5, 2 and 4 


F 


14 


9, 8 and 6 


F 




Text layout 20. 





The method of the present invention searches the fault 
detection set matrix for failed dependency sets that contain 
the same test points (step SI 43), and creates the fault 
isolation set matrix (step S150). The fault isolation matrix is 
provided in Text Layout 21. 



FAULT ISOLATION MATRIX 



Fault Set Includes Code 

Isolation Set Test Points Block 



1 2 2 

2 8 8 
Text Layout 21. 



TEST POINT MAPPING MATRIX 
Test Point Block of Code 30 



35 



40 
45 
50 



RUN TIME PASS/FAIL MATRIX 
Test Point Pass/Fail 



60 In the present example, test points 2 and 8 are found to be 
conclusive faults. Test points 5 and 8, on the other hand, 
have failed status but are not found to necessarily be 
conclusive errors. The method of the presently preferred 
embodiment, as applied to the present example, cannot 

65 determine with certainty whether test points 5 and 9 are 
conclusive failures or whether the failures of test point 5 and 
9 are propagated from test points 2 and 8, respectively. In an 
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embodiment where a masked fault isolation matrix is gen- Computer Aided Design (CAD) package and formatted into 

e rated as an output, test points 5 and 9 are output in the either Electronic Data Interchange Format (ED IF) or VHDL 

masked fault isolation matrix as non-conclusive faults. The format, for example. Both EDIF and VHDL are electrical 

masked fault isolation matrix for the present example is set design industry standard descriptive languages. Both EDIF 

forth in Text Layout 22. 5 anc j VHDL provide formats that can be used with a wide 

variety of tools. Although an electrical CAD package is 
presently preferred for capturing the software block diagram 
(flow chart), other specific design packages can be made to 
save the design in EDIF or VHDL formats through the use 
10 of data file translators. 

As presently embodied, the EDIF and VHDL files contain 
all of the descriptive detail necessary to document and 
model most any computer software design under evaluation. 
Commercially available electronic industry tools can be 
15 used with the basic EDIF or VHDL file to perform diag- 

A key advantage associated with the method of the nostic analysis and to identify, for example, the testability of 

present invention is a reduction in the amount of time and the software design. When implementing such commercially 

cost typically required in implementing software debugging, available electronic industry tools, the setup and application 

verification and validation. Because the program code is of software test points generally very closely corresponds to 

modeled with a knowledge-based reasoning approach, 20 the setup and application of hardware test points, 

which is derived from the software design itself, prior-art The commercially available electronic industry tools can 

requirements of spending great effort in evaluating each line be configured, in accordance with the present invention, to 

of code in connection with algorithm studies, code inspec- perform the software analysis in a similar way to that of the 

tions and independent code analysis is substantially attenu- conventionally- implemented hardware analysis. Two 

ated.The reduction in time and cost is achieved through the 25 ex les of these ana i ytica i t00 ^ which accept EDIF 

modeling of the actual functionality of the code. formatted input files, are the McDonnel Douglas tool called 

In an embodiment where a high-level of automation is AUTOTEST and the U.S. Army TMDE tool called the 

implemented into the method of the present invention, a Diagnostic Analysis and Repair Tool Set (DARTS), 

real-time fault detection and isolation algorithm is used in ™ , . - , - . ♦ • . • j 

conjunction with the functional model of the software to 30 deselection of the software test points, in accordance 

perform diagnostics during the execution of the software ^ th the P* 8 "* "venUon, can be performed during the 

code. Real-time fault detection tools are presently commer- diagnostic analysis of the program code using an EDIF or 

daily available for hardware verification and validation V™* 1 - based analytical tool, such as AUTOTEST or 

application. These real-time fault detection tools for hard- DARTS. These commercially available tools can help the 

ware can be adapted to and applied to software code 35 design engineer select test points which will enhance the 

debugging, verification and validation procedures in accor- testability of the software design, thereby improving the 

dance with the present invention. fault detection and isolation statistics of the present inven- 

Since the method of the present invention comprises a tion - Both tne AUTOTEST and DARTS tools are used to 

structural modeling approach, rather than functional, the document and improve the testability of electronic designs, 

method can be incorporated into commercially available 40 The process diagrams shown in FIGS. 6-10 are compatible 

Hardware Definition Languages, for example, such as Very with the above-mentioned commercially available diagnos- 

High Scale Integrated Circuit Hardware Definition Lan- tic tools. 

guage (VHDL), which is a common digital microcircuit After a pass or a fail status is assigned to each test point 

design and simulation language. The incorporation of VHDL during the program execution, a resulting test point data file 

into the method, in accordance with one embodiment of the 45 ^ processed by a search algorithm which uses the knowl- 

present invention, can allow for simultaneous modeling of c d ge base (functional model) of the software design as a 

hardware, firmware and embedded software. reference, In this embodiment, a dependency comparison of 

The method of the present invention can be extended to pass/fail test point data is performed which enables a search 

include a class of software design and/or drawing tools, to be conducted to eliminate possible failure locations. For 

which are adapted to create and/or accept block diagrams 50 example, the system would conclude that a block in the 

drawn on a Personal Computer (PC) display by a user, and design diagram (flow chart) must be functioning properly if 

which are adapted to "capture" the design. An extension of its inputs are pass and its outputs are pass. By mapping the 

the method of the present invention encompasses a software effects of pass and fail data against the functional model of 

application, for example, which is adapted to receive a block the software code, the process of the present invention can 

diagram, drawn on a PC by a user, and to automatically 55 isolate to the list of possible failures for a given test point 

convert the block diagram into a dependency set matrix. The data file. Since a dependency relationship is naturally 

drawing software application, in one embodiment, is defined within the software flow chart interconnections, it is 

adapted to save the file in a format that delineates the not necessary that every data point in the system diagram be 

connectivity between blocks and the respective inputs and instrumented. 

outputs and locations of test points. A high level drawing 60 n j s no ted that the process exemplified in steps S129 to 

tool, which is adapted to facilitate automatic building of the S153 (FIGS. 3fc-3t/), for example, are not intended to be 

dependency set matrix, can save additional time and can limiting and may be implemented in other ways or with 

extend the utility of the present invention to very large other means other than the computer system 20 of FIG. 1. An 

programs. important aspect of the inventive process is that a pass/fail 

In accordance with one embodiment of the present inven- 65 test point data file for the software code being tested is 

tion, once a block diagram representation of the software generated, and that the test point data file is analyzed using 

design is complete, the block diagram is captured within a a knowledge-based reasoning approach, which preferably 
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comprises a functional model of the software code. executing the computer program on a computer to thereby 

The knowledge-based reasoning approach for software generate actual values for the plurality of test points; 

debugging, verification and validation, in accordance with comparing the expected values for the plurality of test 

the present invention, may be used beyond the development, points with the actual values for the plurality of test 

performance monitoring, diagnostics, and testing phases of 5 points; 

the software product. The process of the present invention identifying a plurality of failed test points, each failed test 

can provide a foundation for performance monitoring during point having an actual value which does not correspond 

operation throughout the life cycle of a software module, with an expected value for the test point; 

thereby enabling users to verify the operation of the software locating at least one source failure test point in the 

at any time during the operational life of the software 10 plurality of failed test points, using the ranked group of 

product. test points, the at least one source failure test point 

The process of the present invention may be expanded to being an earliest failed test point, in the order of 

incorporate automated corrective action through the use of execution and data dependency, among the ranked 

autocode generation, such as disclosed in the closed-loop group of test points. 

corrective action methodology illustrated in FIG. 10. 15 2. The method as recited in claim 1, wherein the ranking 

The method of the present invention can be extended to ste P includes a step of defining at least one fault propagation 

most if not all known forms of software code generation P atn - 

where the software design can be represented as a functional 3. The method as recited in claim 1, wherein: 

block diagram, and can provide a universal environment for the plurality of failed test points corresponds to a single 

debugging software programs that contain subprograms 20 ranked group of test points, the single ranked group of 

written in different languages. Although real-time cross test points defining a fault dependency set. 

compilers are presently in frequent use, modern and future 4. The method as recited in claim 1, wherein: 

environments, such as Internet applications, will likely the plurality of failed test points corresponds to a plurality 

increasingly incorporate multiple code sources for imple- 0 f ranked groups of test pomtSj eacn ranked of 

menting cross platform applications. For example, one high 25 les t points defining a separate fault dependency set. 

level program may be written in Java for multi-hardware 5 . xh e method as set forth in claim 4, wherein the step of 

platform compatibility on the Internet, but the Java mother locating at least one source failure test point in the plurality 

program may contain embedded sibling programs that of failed test points comprises a step of locating at least one 

execute in different native codes, such as Fortran, C, C++, failure tes t point in the plurality of failed test points, 

Basic, etc., once the Java program arrives at its Internet 3Q us i n g the plurality of ranked groups of test points, 

destination. 5 me thod as set forth in claim 1, wherein the step of 

Although an exemplary embodiment of the invention has generating expected values for the plurality of test points 

been shown and described, many other changes, modifica- comprises a step of defining a reference test point data file, 

tions and substitutions, in addition to those set forth in the the reference test point data file storing the expected values 

above paragraphs, may be made by one having ordinary skill 35 for the plurality of test points. 

in the art without necessarily departing from the spirit and 7. The method as set forth in claim 6, wherein the step of 

scope of this invention. executing the computer program on a computer to thereby 

What is claimed is: generate actual values for the plurality of test points com- 

1. A method of locating a source failure test point among prises a step of storing output values of test points in the run 

a plurality of test points in a computer program, the source 40 lime test point data file. 

failure test point having a greatest probability of being an 8. The method as set forth in claim 7, wherein the step of 
originating source of failure among the plurality of test comparing the expected values for the plurality of test points 
points, the method comprising the following steps: with the actual values for the plurality of test points corn- 
ranking the plurality of test points in the computer pro- prises a step of correlating the reference test point data file 

gram in accordance with an order of execution and a 45 with the run time test point data file. 

data dependency of each of the plurality of test points, 9. The method as set forth in claim 8, wherein the step of 

to thereby define a ranked group of test points, the step identifying a plurality of failed test points comprises the 

of ranking comprising: following steps: 

grouping lines of code of the computer program into defining a run time pass/fail matrix, the run time pass/fail 

functional blocks; 50 matrix being adapted to store the plurality of failed test 

identifying inputs and outputs for each functional points; and 

block; entering the plurality of failed test points in the run time 

creating a block diagram showing the interconnectivity pass/fail matrix. 

of the functional blocks; 10. The method as set forth in claim 9, wherein the step 

identifying dependency sets in the block diagram, each 55 of locating at least one source failure test point in the 

dependency set (fault dependency set) defining a plurality of failed test points comprises a step of determining 

fault propagation path, which indicates a flow of data which dependency sets contain failed test points, using the 

or program operation through a number of the func- dependency set matrix. 

tional blocks; U. The method as set forth in claim 10, wherein the step 

defining a run time test point data file, the run time test 60 of locating at least one source failure test point in the 

point data file storing output values of test points plurality of failed test points comprises a step of locating the 

during execution of the computer program; and at least one source failure test point in the plurality of failed 

defining a dependency set matrix, the dependency set test points, using the dependency sets in the dependency set 

matrix defining at least one dependency set; matrix that were determined to contain failed test points, 

generating expected values for the plurality of test points 65 12. The method as set forth in claim 10, wherein the step 

for an expected, proper-operation execution of the of locating at least one source failure test point in the 

computer program; plurality of failed test points comprises the following steps: 
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defining a fault detection set matrix, the fault detection set 
matrix being adapted to store dependency sets therein; 

storing the dependency sets that contain failed test points 
in the fault detection set matrix; 

determining which dependency sets in the fault detection 5 
set matrix contain the same failed test points; and 

locating the at least one source failure test point in the 
plurality of failed test points using the dependency sets 
that contain failed test points. 1Q 

13. The method as set forth in claim 12, wherein the step 
of locating at least one source failure test point in the 
plurality of failed test points comprises the following steps: 

determining whether a number of test points corresponds 
to a number of functional blocks; and 15 

determining a functional block that corresponds to each 
test point, upon a determination that the number of test 
points does not correspond to a number of functional 
blocks, 

14. A method of locating a source failure test point among 20 
a plurality of test points in a computer program, the source 
failure test point having a greatest probability of being a 
source of failure in the computer program, the method 
comprising the following steps: 

ranking the plurality of test points in accordance with an 25 
order of data dependency of the test points, to define a 
ranked group of test points, wherein a highest ranked 
test point in the ranked group of test points is not 
dependent on any test point in the ranked group of test 
points, and wherein a lowest ranked test point in the 30 
ranked group of test points is ultimately dependent on 
all of the test points in the ranked group of test points, 
wherein the step of ranking comprises: 
grouping lines of code of the computer program into 

functional blocks; 35 
identifying inputs and outputs for each functional 

block; 

creating a block diagram showing the interconnectivity 
of the functional blocks; 

identifying dependency sets in the block diagram, each 40 
dependency set defining a fault propagation path, 
which indicates a flow of data or program operation 
through a number of the functional blocks; 

defining a run time test point data file, the run time test 
point data file storing output values of test points 45 
during execution of the computer program; and 

defining a dependency set matrix, the dependency set 
matrix defining at least one dependency set; 
generating expected values for the plurality of test points 

for an expected, proper-operation execution of the 50 

computer program; 
executing the computer program on a computer to thereby 

generate actual values for the plurality of test points; 
comparing the expected values for the plurality of lest 55 

points with the actual values for the plurality of test 

points; 

identifying a plurality of failed test points, each failed test 
point having an actual value which does not correspond 
with an expected value for the test point; and 60 

locating a source failure test point in the ranked group of 
test points, the source failure test point having a highest 
ranking among failed test points in the ranked group of 
test points. 

15. The method as set forth in claim 14, wherein the step 65 
of generating expected values for the plurality of test points 
comprises a step of defining a reference test point data file, 
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the reference test point data file storing the expected values 
for the plurality of test points. 

16. The method as set forth in claim 15, wherein the step 
of executing the computer program on a computer to thereby 
generate actual values for the plurality of test points com- 
prises a step of storing output values of test points in the run 
time test point data file. 

17. The method as set forth in claim 16, wherein the step 
of comparing the expected values for the plurality of test 
points with the actual values for the plurality of test points 
comprises a step of correlating the reference test point data 
file with the run time test point data file. 

18. The method as set forth in claim 17, wherein the step 
of identifying a plurality of failed test points comprises the 
following steps: 

defining a run time pass/fail matrix, the run time pass/fail 
matrix being adapted to store the plurality of failed test 
points; and 

entering the plurality of failed test points in the run time 
pass/fail matrix. 

19. The method as set forth in claim 18, wherein the step 
of locating at source failure test point in the ranked group of 
test points comprises the following steps: 

determining which dependency sets contain failed test 
points, using the dependency set matrix; and 

locating the at least one source failure test point in the 
plurality of failed test points, using the dependency sets 
in the dependency set matrix that were determined to 
contain failed test points. 

20. The method as set forth in claim 18, wherein the step 
of locating at source failure test point in the ranked group of 
test points comprises the following steps: 

defining a fault detection set matrix, the fault detection set 

matrix being adapted to store dependency sets therein; 
storing the dependency sets that contain failed test points 

in the fault detection set matrix; 
determining which dependency sets in the fault detection 

set matrix contain the same failed test points; and 
locating the source failure test point in the ranked group 

of failed test points using the dependency sets that 

contain failed test points. 

21. A method of selecting a source failure test point from 
a plurality of test points in a computer program, the source 
failure test point having a highest probability relative to 
other test points of being a source of failure in the computer 
program, the method comprising the following steps: 

forming a test point group from the plurality of test points, 
wherein test points in the test point group depend from 
one another to define a data flow path; 

ranking the test points in the test point group in accor- 
dance with an order of execution and data dependency 
of the test points, wherein the step of ranking com- 
prises: 

grouping lines of code of the computer program into 

functional blocks; 
identifying inputs and outputs for each functional 

block; 

creating a block diagram showing the interconnectivity 
of the functional blocks; 

identifying dependency sets in the block diagram, each 
dependency set defining a fault propagation path, 
which indicates the data flow path through a number 
of the functional blocks; 

defining a run lime test point data file, the run time test 
point data file storing output values of test points 
during execution of the computer program; and 
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defining a dependency set matrix, the dependency set 
matrix defining at least one dependency set; 

generating expected values for the plurality of test points 
for an expected, proper-operation execution of the 
computer program; 5 

executing the computer program on a computer to thereby 
generate actual values for the plurality of test points; 

comparing the expected values for the plurality of test 
points with the actual values for the plurality of test 
points; 10 

identifying a plurality of failed test points, each failed test 
point having an actual value which does not correspond 
with an expected value for the test point; and 

determining which of the plurality of failed test points 
belong to the test point group; and 15 

locating the source failure test point by finding a failed 
test point within the test point group which does not 
depend from any other failed test points within the test 
point group and which is earliest in execution in the 
data flow path relative to the other failed test points 20 
within the test point group. 

22. A method of selecting a source failure test point from 
a plurality of test points in a computer program, the source 
failure test point having a highest probability relative to 
other test points in the computer program of being a source 25 
of failure, the method comprising the following steps: 

providing a plurality of test points in a computer program, 
including: 

grouping lines of code of the computer program into 

functional blocks; and 
identifying inputs and outputs for each functional 

block; 

defining at least one fault propagation path, the at least 
one fault propagation path associating at least two of 
the plurality of test points in an order of data flow and 35 
data dependency within the computer program includ- 
ing: 

creating a block diagram showing the interconnectivity 
of the functional blocks; 

identifying a plurality of fault propagation paths in the 
block diagram, each fault propagation path indicat- 
ing a flow of data or program operation through a 
number of the functional blocks; 

defining a run time test point data file, the run time test 
point data file storing output values of test points 
during execution of the computer program; and 

defining a dependency set matrix, the dependency set 
matrix defining at least one fault propagation path; 
generating expected values for the plurality of test points 5Q 

for an expected, proper-operation execution of the 

computer program; 
executing the computer program on a computer to thereby 

generate actual values for the plurality of test points; 
comparing the expected values for the plurality of test 55 

points with the actual values for the plurality of test 

points; 

identifying a plurality of failed test points, each failed test 
point having an actual value which does not correspond 
with an expected value for the test point; and eo 

finding, for the at least one fault propagation path, the 
source failure test point which is earliest, relative to 
other failure test points in the at least one fault propa- 
gation path, in an order of data flow and data depen- 
dency. 65 

23. The method as set forth in claim 22, wherein the step 
of generating expected values for the plurality of test points 



45 



comprises a step of defining a reference test point data file, 
the reference test point data file storing the expected values 
for the plurality of test points. 

24. TTie method as set forth in claim 22, wherein the step 
of executing the computer program on a computer to thereby 
generate actual values for the plurality of test points com- 
prises a step of storing output values of test points in the run 
time test point data file. 

25. The method as set forth in claim 24, wherein the step 
of comparing the expected values for the plurality of test 
points with the actual values for the plurality of test points 
comprises a step of correlating the reference test point data 
file with the run time test point data file. 

26. The method as set forth in claim 25, wherein the step 
of identifying a plurality of failed test points comprises the 
following steps: 

defining a run time pass/fail matrix, the run time pass/fail 
matrix being adapted to store the plurality of failed test 
points; and 

writing the plurality of failed test points to the run time 
pass/fail matrix. 

27. The method as set forth in claim 26, wherein the step 
of finding the source failure test point comprises a step of 
determining which fault propagation paths contain failed test 
points, using the dependency set matrix. 

28. The method as set forth in claim 27, wherein the step 
of finding the source failure test point further comprises a 
step of locating the at least one source failure test point in the 
plurality of failed test points, using the fault propagation 
paths in the dependency set matrix that were determined to 
contain failed test points. 

29. The method as set forth in claim 26, wherein the step 
of finding the source failure test point comprises the fol- 
lowing steps: 

defining a fault detection set matrix, the fault detection set 
matrix being adapted to store fault propagation paths 
therein; 

storing the fault propagation paths that contain failed test 

points in the fault detection set matrix; 
determining which fault propagation paths in the fault 

detection set matrix contain the same failed test points; 

and 

locating the at least one source failure test point in the 
plurality of failed test points using the fault propagation 
path that contain failed test points. 

30. A method of determining a source failure test point 
from a plurality of test points in a computer program, the 
source failure test point having a highest probability relative 
to other test points in the computer program of being a 
source of failure, the method comprising the following 
steps: 

determining a sequential flow of data among a plurality of 
test points in a computer program, including: 
grouping lines of code of the computer program into 

functional blocks; 
identifying inputs and outputs for each functional 

blocks; and 

creating a block diagram showing the interconnectivity 
of the functional blocks; 
ranking the plurality of test points, using the determined 
sequential flow of data, in an order of an earliest test 
point in the determined sequential flow of data to a last 
test point in the determined sequential flow of data, to 
thereby generate a ranked set of test points, the step of 
ranking including: 

identifying dependency sets in the block diagram, each 
dependency set indicating a flow of data or program 
operation through a number of the functional blocks; 
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defining a run lime point data file, the run time test 
point data file storing output values of test points 
during execution of the computer program; and 
defining a dependency set matrix, the dependency set 
matrix defining at least one dependency set; 
generating expected values for the plurality of test points 
for an expected, proper-operation execution of the 
computer program; 
executing the computer program on a computer to thereby 
generate actual values for the plurality of test points; 
comparing the expected values for the plurality of test 
points with the actual values for the plurality of test 
points; 

identifying a plurality of failed test points from among the 
plurality of test points, each failed test point having an 
actual value which does not correspond with an 
expected value for the test point and thereby indicating 
an erroneous program operation or result of the com- 
puter program at the failed test point; and 

determining a failed test point of the plurality of failed test 
points which ranks earliest among failed test points in 
the ranked set of test points, the earliest -ranked failed 
test point being the source failure test point, 

31. The method as set forth in claim 30, wherein the step 
of generating expected values for the plurality of test points 
comprises a step of defining a reference test point data file, 
the reference test point data file storing the expected values 
for the plurality of test points. 

32. The method as set forth in claim 31, wherein the step 
of executing the computer program on a computer to thereby 
generate actual values for the plurality of test points com- 
prises a step of writing output values of test points to the run 
time test point data file. 

33. The method as set forth in claim 32, wherein the step 
of comparing the expected values for the plurality of test 
points with the actual values for the plurality of test points 
comprises a step of correlating the reference test point data 
file with the run time test point data file. 

34. The method as set forth in claim 33, wherein the step 
of identifying a plurality of failed test points comprises the 
following steps: 

defining a run time pass/fail matrix, the run time pass/fail 
matrix being adapted to store the plurality of failed test 
points; and 

writing the plurality of failed test points to the run time 
pass/fail matrix. 

35. The method as set forth in claim 34, wherein the step 
of locating at least one source failure test point in the 
plurality of failed test points comprises the following steps: 

defining a fault detection set matrix, the fault detection set 

matrix being adapted to store dependency sets therein; 
storing the dependency sets that contain failed test points 

in the fault detection set matrix; 
determining which dependency sets in the fault detection 

set matrix contain the same failed test points; and 
locating the at least one source failure test point in the 

plurality of failed test points using the dependency sets 

that contain failed test points. 

36. The method as set forth in claim 34, wherein the step 
of locating at least one source failure test point in the 
plurality of failed test points comprises the following steps: 

determining which dependency sets contain failed test 
points, using the dependency set matrix; and 

locating the at least one source failure lest point in the 
plurality of failed test points, using the dependency sets 



in the dependency set matrix that were determined to 
contain failed test points. 

37. A method of locating a source failure test point among 
a plurality of test points, the source failure test point having 
a greatest probability of being a source of failure in a 
computer program, the method comprising the following 
steps: 

placing a plurality of test points into a computer program; 
determining an order of data flow among the plurality of 
test points, the order of data flow defining at least one 
data propagation path among the plurality of test 
points, the step of determining including: 
grouping lines of code of the computer program into 

functional blocks; 
identifying inputs and outputs for each functional 
block; 

creating a block diagram showing the interconnectivity 

of the functional blocks; 
identifying data propagation paths in the block 
diagram, each data propagation path defining a fault 
propagation path, which indicates a flow of data or 
program operation through a number of functional 
blocks; 

defining a run time test point data file, the run time test 
point data file storing output values of test points 
during execution of the computer program; and 
defining a dependency set matrix, the dependency set 
matrix defining at least one data propagation path; 
generating expected values for the plurality of test points 
for an expected, proper-operation execution of the 
computer program; 
executing the computer program on a computer to thereby 
generate actual values for the plurality of test points; 
comparing the expected values for the plurality of test 
points with the actual values for the plurality of test 
points; 

identifying at least two failed test points from the plurality 
of test points, each failed test point having an actual 
value which does not correspond with an expected 
value for the test point; and 
associating the at least two failed test points with the at 

least one data propagation path; and 
locating the source failure test point from among the at 
least two associated failed test points by selecting a 
failed test point which is earliest in the at least one data 
propagation path. 

38. The method as set forth in claim 31, wherein the step 
of generating expected values for the plurality of test points 
comprises a step of defining a reference test point data file, 
the reference test point data file storing the expected values 
for the plurality of test points. 

39. The method as set forth in claim 38, wherein the step 
of executing the computer program on a computer to thereby 
generate actual values for the plurality of test points com- 

55 prises a step of storing output values of test points in the run 
time test point data file. 

40. The method as set forth in claim 39, wherein the step 
of comparing the expected values for the plurality of test 
points with the actual values for the plurality of test points 

60 comprises a step of correlating the reference test point data 
file with the run time test point data file, 

41. The method as set forth in claim 40, wherein the step 
of identifying at least two failed test points comprises the 
following steps: 

defining a run lime pass/fail matrix, the run lime pass/fail 
matrix being adapted to store the plurality of failed test 
points; and 
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entering the plurality of failed test points in the run time 
pass/fail matrix. 

42, The method as set forth in claim 41, wherein the step 
of associating the at least two failed test points with the at 
least one data propagation path comprises a step of deter- 
mining which data propagation paths contain failed test 
points, using the dependency set matrix. 
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43. The method as set forth in claim 42, wherein the step 
of locating the source failure test point comprises a step of 
locating the source failure test point, using the data propa- 
gation paths in the dependency set matrix that were dctcr- 
5 mined to contain failed test points. 

***** 
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